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ABSTRACT 

The purpose of this study was to develop the 
Johnson-Neynan Procedure (JH-Procedure) appropriate to multiple 
groups and covariables, and demonstrate its use in the analysis of 
group differences • A sequence of significance tests vhich makes it 
possible to identify the most parsimonious analysis of group 
differences appropriate to a given set of data is presented. In 
addition, there is an orignial mathematical derivation of the 
JN-Procedure appropriate to multiple groups and covariates, and a 
demonstration of its superiority to ABCO?A vhen group regressions are 
heterogeneous » (Author) 



us DEPARTMENT OF HEALTH, 
EDUCATION & WELFARE 
NATIONAL INSTITUTE OF 
EOUCATION 



An Alternative to Ancova When Group Regressions 
Are Heterogeneous : 
The Generalized Johnson-Neyman Procedure 
by 



;:c\oTxr/rr: rit?^ -^'Zl Fred For ster 

IVfNr.r'"pS:iT.?o"rc'."roptN.i:s Portland Public Schools / 

631 N.E. Ciackamas street 
fcoucAT.oNPos.T.oNOKPoucy PortUttd, Oregon 

97208 

Presented at the 1975 Annual Meeting of the American Educational Rese&rch 
|Nr\ Association, Washington, March 30 to April 3, 1975. 

O 



In a variety of research situations it is desirable to increase the 
precision of the analysis of differences among treatments by using one or 
more covariate scores. A reoccuring (though often undetected) problem which 
occurs in this type of analysis is the presence of heterogeneous group 
regressions which indicate that the effect of the treatments is related to 
specific characteristics of the students. If analysis of covariance 
(ANCOVA) is mistakenly applied when the group regressions are heterogeneous, 
it can lead to erroneous conclusions since the Type II error of the analysis 
increases as the degree of heterogeneity increases (Peckham, 1968). An 
alternate procedure has been developed to analyze heterogeneous group 
regressions research situations involving two groups and two covariates 
(Johnson and Neyman, 1936). While some efforts have been made to extend 
QQ this procedure to accomodate multiple groups and covariates (Ableson, 1953; 

4/5 Potthoff, 1964), the practical application of this technique has been limit- 

^IsH groups and one or two covariates. The problem addressed in this 

paper is the specification of a methodology for efficiently determining when 
^ group regressions are heterogeneous, and for applying the Johnson-Neyman 

procedure regardless of the number of groups or covariates in the analysis. 
This methodology will be discussed in relation to the analysis of an experi- 



ment in human relations training for classroom teachers. 
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THEORETICAL DEVELOPMENT 

The approach followed here is similar to that proposed by Pearson, 
Neyman and their students (Pearsott and Neyman, 1928; Kolodziejczyk, 1935; 
Johnson and Jackson, 1959). Two alternative regression models are proposed 
and the maximum likelihood method is used to identify the sum of squares 
and degrees of freedom appropriate to each model. By subtracting the smaller 
sum of squares from the larger and forming the appropriate F-ratio, it is 
possible to develop a statistical test between the models. An important 
aspect of this procedure is the sequencing of these statistical tests of 
hypotheses so that the minimum necessary effort is expended. The system of 
tests described below has been developed to achieve this esult. 

Statistical Test I - Do Any Significant Group Differences Exist 

This test is intended to provide a go/no-go criteria for continuing with 
the analysis. If the F-ratio is not statistically significant then the 
analysis should be terminated, otherwise it is worthwhile to proceed to 
additional analyses. 

The two regression models employed for this test are: 

Hj: « B Xj, Y2 = B X2, . . . , Yj^ « B Xj^ 

(null hypothesis: all groups have the same regression 
equation) and 

H^: Yj » BiXj Yj - B2X2» * • • • f \ " ^k^k 

(alternative hypothesis: the groups have different re- 
gression equations) 
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Using the maximum likelihood method (Forster, 1974) it is possible 
to show that the sum of squares appropriate to is; 



i-1 J-1 



where _ N. 
R 



i 



'^'^ 



•^i" Tl ""i'^ 
j-i 

^i-^i ^i 

k 

d£ - N. - k(n+l) 
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The appropriate F-ratio is: (§2 . / (jf^ . jf^) 
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Statistical T^st II - Do The Groups Have Heterogeneous Regressions 

The two regression models employed ior this test are as defined 
before, and 



(null hypothesis: group regressions are homogeneous although 
group means may be different) 



where 
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df_ = ZH N. - (km) 
i=1 . 



The appropriate P-ratic is: 



The decision rule for this test is: 

(1) when is statistically significant conclude group 
regressions are heterogeneous. 

(2) when is not statistically significant use ANCOVA. 
Interestingly, if ANCOVA is the indicated procedure, the 

necessary sums o2 squares are already available: 

The Generalized Johnson**Neyman Procedure 

When heterogeneous group regressions have been Identified, it is 
necessary to qualify the effectiveness of a treatment on the basis of specific 
characteristics of subjects as reflected by the covariables. 
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The two regression models for this test are as defined before 

and 

(null hypothesis: the treatments are equally effective 
for W, a specified set of covariate scores) 
The deteiroination of the siara of squares appropriate to H. is more 
involved than the previous examples. Since the second part of H^^ is 
identical to that for H^, a similar equation results for thr- sum of 



squares: 



N 

iTi jTi i=.1 ^ ^ 



The calculation of requires the introduction of the constraints 
W = . . . = through LaGrangian multipliers, resulting in 

the following system of equations: 

^1 =^ c^^V' 



w 



Returning to the basic constraint that W(B^ - B^) = 0 where B^ P 

S ^ 0 and imposing an addi^^ional constraint (for simplicity) that 
J 

K 

2 C. = 0 these equations can be expressed as 
i=1 ^ 
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It is possible to solve for C = iT^ K where U and K denote the right side 
matrix and left side vector respectively. This makes it possible to solve 
for B. and the appropriate sum of squares: 

^Jn ^ 2 j 'X 

i=1 j=1 i=1 

N. 



k 1 



i=1 j=1 ^'J i=1 ^ ^ i=1 ^ ^ 



J 

k 



i=1 

The appropriate P-ratio is: 

The decision rule for this test is: 

(1) When P. is statistically significant conclude that there 
^ ' jn 

is a significant difference among treatments for individuals vith 
the characterlsitics represented by the specified set of covariate 
scores « 

(2) When F is not statistically significant conclude that the 
treatments do not differ significantly for the individuals 
vith the specified set of covariate scores. 
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The results of the generalized JN-Procedure can be uned to classify 
ea ch Individual to the treatment group predicted to provide maximum benefit 
based on the specific set of covarlable scores* Indlvldualr not predicted 
to benefit significantly more from any of the methods can be asslged to fill 
out the necessary groups. 

The use of these tests Is summarized In Figure 1 showing the sequence 
of tests and the decisions to be made following each step* 



APPLICATION 

Project Description 

The data analyzed to demonstrate the generalized JN-Procedure were 
drawn from a study by Bowers and Soar (Bowers and Soar, 1960) . This study 
was designed to determine the impact of Intensive laboratory training in 
human relations on the classroom performance of teachers* To collect data 
related to the goals of the study teachers completed several scales fran 
the Minnesota Multiphasic Personality Inventory (MMPI) including the scales 
shown below: 



(a 
(b 
(c 
(d 
(e 
(f 
(g 
(h 
(1 
(j 
(k 
(1 

(m: 

(n 

(o: 
(p 

(q 
(r 



Hypochrondrlasls scale ^Hs) , 

Depression scale (D) , 

Hysteria scale (H), 

Psychopathic devlption scale (Pd) , 

Masculine-Feminine interest scale (MF) , 

Paranoia scale (Pa), 

Psychasthenia scale (Pt) , 

Schizophrenia scale (Sc) , 

Hypomania scale (Ma) , 

Social introversion scale (SI) , 

Lie scale (L) , 

F validity scale (F) , 

K validity scale (K) , 

Anxiety scale (A) , 

Repression scale (R) , 

Ego-strength scale (Es) , 

Hostility scale (Ho) , 

Pharisaic-virtue scale (Pv) . 
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Conclude 
no significant 
differences 

among 
treatments 



JOHNSON-NEYMAN 
Procedure 



Is there 
a difference 
among treatments 
for this individual 
pmbination 

(H. vs 

■ jn 



MCOVA 




YES 



Place this 
individual 

in any 

treatment 




YES 



Place this 
individual in 
the preferred 
treatment 



Conclude 
no treatment 
superior 
to others 



Conclude 
one or more 
treatments 
superior to 
others 



FIGURE 1 
Sequence of Tests for the 
Generalized Johnson-Neyman Procedure 
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In addition, teachers were asked to report on the number of activities 
vhich they planned cooperatively vith other teachers (SR), the Observation 
Schedule and Record (0SCAR2E) was used to assess the social structures 
observed in each teachers classroom on a pre and post treatment basis (SSp^e* 
SSpost), and the Russell Sage Social Relations observation scale was used 
in each classroom. Sixty-two elementary school teachers were recruited to 
participate in the Bowers and Soar project. Of these, eight dropped out of 
the study and three others taught special education classes and were not 
included in the analysis. This left a total of fifty-one teachers in the 
study, twenty-eight in the control group, and twenty-three in the experimental 
group. Since participation in the experimental group required attendance at 
a three-week workshop either immediately after school adjourned for the 
summer or immediately before school opened in September, some teachers were 
not able to participate in either workshop. Consequently, assignment to 
the treatments was noi.-random. The experimental group was composed entirely 
of teachers able and willing to participate in the workshop, while the control 
group included some teachers unable or unwilling to participate in the work- 
shop. In spite of this, the comparison of means and distributions for 
personal history data, pretraining measures, and school socioeconomic data 
as reported by Bowers and Soar did not reveal any significant differences 
between the groups. (Bowers and Soar, 1960, pp. 49-50) 

During the spring semester 1959, each teacher was asked to complete 
inventory and questionnaire data. In addition, the 0SCAR2E was used during 
the spring semester to complete the pretraining data. 

During the summer, teachers in the experimental group attended one of 
two workshops on Human Relations Training. These workshops included three 
primary activities: 
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(1) Theory sessions, using a relatively traditional lecture 
format^ 

(2) skill practice, using role play, group discussion, and 
related activities, and 

(3) . training group, using unstructured group discussion. 

Reading lists and a collection of reading materials vere made available 
to participants, but no assignments were made. The following spring semester 
follow-up data was gathered of approximately the same nature as the pretrain- 
ing data. Although a number of hypotheses were presented by Bowers and Soar, 
this reanalysis of their data focused on the following hypothesis proposed 
in their original study: 

There are no differences between teachers who have experienced 
intensive laboratory training in human relations and teachers 
who have not experienced such training with respect to the 
Social Structures of the classroom. 

Analyses of Data 

Since regression analyses have been shown to >e adversely affected by 
heterogeneity of variance among groups with respect to the covariables used 
in the analysis, high intercorrelations among the covariables, and departure 
from a linear relationship between the dependent variable and the covariable, 
preliminary tests were performed to determine whether any or all of these 
conditions might exist. These tests indicated that there were no major 

^'v J J 

departures from homogeneity of variance between the groups when the small 
sample sizes were taken into account. In addition, the similarity of values 
for the means and me ''iana indicated that the distributions of the variables 
were not skewed significantly. 
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The data were screened to identify pairs of covariables which were 
so highly correlated with the SS posttraining scores as to introduce 
instability into the group regression equations. Using a test proposed 
by Lord and Novick (Lord and Novick, pp 266-269), it was decided to 
drop several covariables fix)m the analysis includ Xt. Sc, Pv, 
SI, Hs, R, D, Es and Ma scales of the MMPI. 

Scatterplots were developed for each group contrasting each potential 

covariable with SS ^ to identify departure from linearity in the data, 
post 

The visual analysis of these graphs did not suggest a significant departure 

from linearity for any of the covariables. A supplementary analysis was 

perforaed to statistically test the de;^ree to which the relationship 

between each covariable and the criterion significantly departed from 

linearity by stepwise adding terms to the regression model: 

y = b^ + b^x + b^x + ... + b^x 

At each step the calculation was made of P-reg indicating the statistical 

significance of the fit and of P-dep indicating the departure from fit. 

The results from this procedure aJ.so indicated there were no significant 

departures from linearity for the covariables with SS^^^^. 

The final screening procedure used was a forward stepwise regression 

analysis for each group, to identify the set of covariables which maximized 

the size of the multiple correlation coefficient inhere SSp^^^ was the 

criterion variable. The sets of maximally effective covariables identified 

for the experimental and control groups had only two members in common, 

SS and Ho. Fbur additional covariables were effective for either one 
pre 
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treatment or the other and were retained in the analysis* The six covariables 
selected for use in this analysis were SSp^^, RSSR, SR, Fa, ^^nd Ho. 

The first test contrasted the null hypothesis that no treatment differ- 
ences exist with respect to means* or regressions with the hypothesis that 
the groups differ with respect to means and regressions (H^ vs. H3). As 
shown in Table 1-A, the F-ratio attained for this test was statistically 
significant (p<.01)5 indicating that the analysis should proceed. 

The F-ratio was statistically significant (p<.05) for the test of the 
hypothesis that there are no group differences with respect to regressions 
versus the hypothesis that the groups differ with respect to means and 
regressions (H2 vs. H3). This result indicated that the group regressions 
were heterogeneous and the JN-Frocedure was the appropriate analysis for 
these data. 

Table 2 summarizes Fjjj for each teacher in the study for whom Fjjj was 
statistically significant. As discussed in the previous analysis, the 
teachers with scores yielding statistically significant F-ratios fall within 
a region of significance, and the remaining teachers have scores which fall 
within a region of non-significance. In comparing the covariable scores for 
the teachers for whom Fj^^ is statistically significant and the regression 
coefficients for the experimental and control groups (as shown in Tables 1-B 
and 2) the effectiveness of the human relations training appears to be 
associated with low pretraining scores on Fa and L. This relationship is 
dramatized by the scores for the two teachers in the experimental group 
(E15 and E22) for whom the treatment is predicted to be inferior to the 
control condition. Those teachers had the highest scores on the RSSR and SR, 
two of the three t^tighest scores scores on Ho, the lowest scores on Fa, and 
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TABLE 1-A 

F Tests For The Generalized Johnson-Nejrman 
Procedure Using The Bomrs-Soar Data 



Number of Groups = 2 

Number of Covariables = 6 



Test 


Source 


Degrees ol 
Preedora 


Sums of 
Squares 


Mean 
Square 


P- ratio 


vs. 


2 2 


7 


591.615 


55.944 


5.471** 


vs. 




6 


278.654 


46.459 


2.881* 




^/ 


57 


596.450 


15.^17 





TABLE 1-B 

Unrestricted Group Regression Coefficients 



Group 


Mean 


Ooveiriab 


le 






SS 

pre 


SR 


L 


RSSR 


Pa 


Ho 


Experimental 


74.416 


.262 


-.559 


-.025 


-.156 


-. 59 


-.159 


Control 


55.464 


.241 


-.074 


-.227 


.096 


-.51^ 


.255 
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relatlvely lov scores on L. Based on these data, the experimental treatment 
appears to have the greatest effectiveness with teachers who have pretralnlng 
scores on the NMPI scales related to the need for acceptance and approval 
and paranoia y aud who report a low level of student Interaction In the class- 
room, for whom few Instances of student Interaction are observed during the 
RSSR tasks, and who are low In hostility. This conclusion Is consistent 
with expectations that the effectiveness of the treatment would be greater 
for teachers who need help with regard to the level of student Interaction 
in the classroom, who are not hostile, and who demonstrate a need for accep- 
tance and approvals 

The application of ANCOVA to these data yields an F-ratlo of 5.552 with 
one and forty-three degrees of freedom (p .05), an adjusted mean for the 
experimental group of 56.337 and an adjusted mean for the control group of 
53.884. Thus, the null hypothesis would be rejected by both ANCOVA and the 
JN-Procedure. As with the research situation, the JN-Procedure provides 
more Information than ANCOVA concerning the specific Individuals most likely 
to benefit from the treatment. The JN-Procedure Indicates that the treat- 
ment would significantly benefit only fourteen of the fifty-one teachers in 
the study (27.5 per cent). 
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SUMMARY 

The application of the Generalized Johnson-Neyman procedure to the data 
previously reported by Bowers and Soar suggests that the training in hutnan 
relations had its most significant impact for those teachers who reported 
few cooperative activities with their colleagues, who demonstrated a need 
for acceptance, who were low in hostility and whose students did not perform 
wel?v on cooperative projects. The principle advantage of the JN-Procedure 
over ANCOVA in this situation was that it identified the relatively few 
teachers who would be expected to significantly benefit from the treatment. 
This characteristic of the Generalized JN-Procedure has significant impli- 
cations for those researchers and administrators who desire to allocate 
teacher training resources in the most effective way. 
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